Methods and circuits for intrinsic processing of image data within image sensing devices

ABSTRACT

An apparatus comprising an analog photocell adapted to capture light energy incident upon it as an analog signal, a sample-and-hold amplifier coupled to the photocell and adapted to store the analog signal and a digital converter coupled to the amplifier, the converter transforming the analog signal into a digital value, the value proportional to the amount of the light energy.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The invention relates generally to the field of image processing.More specifically, the invention relates to image or motion videocompression.

[0003] 2. Description of the Related Art

[0004] In the current state of the art, image capture devices, thosedevices which represent an environment or scene by electrical signalsthat are proportional to the color and/or intensity of light present inthe scene, are often manufactured and designed using CCD (Charge CoupledDevice) technology. A CCD image capture device utilizes small photocellsto generate electrical signals that are related to the incident lightfrom the scene that strikes the photocells. The imaging device containsa two-dimensional array of such photocells such that a series of signalsacross an entire focused upon scene can be captured and stored. Morerecently, CMOS (Complementary Metal Oxide Semiconductor) imager deviceshave been developed which function to provide the same sort of outputsignals that CCD devices to but often at a lower cost and complexity.Examples of common imaging devices include scanners, motion videocameras and digital still cameras.

[0005] Many of these imaging devices, whether based on CMOS or CCDtechnology, are designed to be compliant with a transmission standardknown as NTSC (National Television Systems Committee). One stricture ofNTSC is that an image should be transmitted not in successive scan rows(of the imager array), but with odd rows transmitted separately from theeven rows. This process of separating odd and even rows is commonlyreferred to as an interlaced scan. An NTSC signal has light intensityinformation encoded as analog voltage levels, color information encodedin the phase and amplitude of a color carrier frequency and so on. Whenan NTSC signal is forwarded for image processing to a computer system,the computer system utilizes a signal converter to transform the analogencoded information into luminance and chrominance digital values forthe transmitted image. The most commonly used luminance-chrominanceformat for the digital representation of images is known as YCrCb (adigital color system referred to as the CCIR (International ConsultativeCommittee on Broadcasting) 601 color space). The conversion from NTSC toYCrCb is serial in nature and due to the serial processing character ofmost commercially available microprocessors. Subsequent image processingis also performed in serial. One notable exception to the predominanceof serial data processing is Intel's MMX(TM) technology based processorswhich use SIMD (Single Instruction Multiple Data) processing. Tocomplement the use of such processors in conjunction with imagingdevices, it would be useful to have parallel processing of the cellsused to capture digital values. Further, a key factor in the practicalapplication of the digital photocell is that the relatively longintegration times of the analog photocell portion allows the use of arelatively slow, but therefore simple method of digitization. For motionvideo, which involves certain inherently serial operations suchper-pixel difference calculations (where the difference between pixelsand/or frames rather than the original values are encoded), it is usefulto implement an architecture that allows such calculations to beperformed on the imaging device rather than strictly through a hostprocessor.

SUMMARY OF THE INVENTION

[0006] What is disclosed is an apparatus comprising an analog photocelladapted to capture light energy incident upon it as an analog signal, asample-and-hold amplifier coupled to the photocell and adapted to storethe analog signal and a digital converter coupled to the amplifier, theconverter transforming the analog signal into a digital value, the valueproportional to the amount of the light energy.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The objects, features and advantages of the method and apparatusfor the present invention will be apparent from the followingdescription in which:

[0008]FIG. 1 illustrates a digital photocell utilized in the invention.

[0009]FIG. 2 is a simplified block diagram of a conventional serialimager.

[0010]FIG. 3 illustrates an architecture for more efficient imagedifferencing.

[0011]FIG. 4 illustrates one embodiment of the invention.

[0012]FIG. 5 illustrates a per-pixel analog difference engine accordingto an embodiment of the invention.

[0013]FIG. 6 illustrates a per-pixel digital difference engine accordingto an embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

[0014] One aspect of the invention involves enhancing each photocellthat is used in the imaging array to capture a scene. Rather than usinga purely analog photocell it may be advantageous, in devicesmanufactured from CMOS technology, to utilize a combination of analogand digital signaling. An analog photocell can be embedded, according toone embodiment of the invention, with conversion circuitry to create adigital photocell. The digital photocell will convert the analog signalgenerated by incident light into a digital code, allowing it to imageprocessed in the digital domain. An array of such digital photocells canbe used, as shown in FIG. 6, to implement a digital image processingsystem on the imaging device.

[0015]FIG. 1 illustrates a digital photocell utilized in the invention.

[0016] An analog photocell 110 captures the light energy incident uponit from the scene being imaged. The analog photocell 110 operatesaccording to an integration time, T, which varies according to ambientlight condition. The integration time is less than the interval neededto saturate the photocell. The charge accumulated at the photocell isinput to a sample and hold amplifier 120. When the photocell dischargesits charge, a counter 140 is reset and begins counting for the nextintegration period. The counter 140 is driven by a voltage controloscillator (VCO) 130. VCO 130 has an input level set by a previouslyacquired charge that has been stored in the sample and hold amplifier120. VCO 130 controls the speed at which the counter 140 increases. Thegreater the light intensity at the analog photocell the faster thecounter 140 will be driven by VCO 130. During an integration period fora particular charge, counter 140 is counting up, and before reset, itsvalue is sent to a register 150. The digital value in register 150,which is also the final value of counter 140, reflects the intensityvalue of the pixel in the previous integration period. For a motionimaging system, register 150 contains the pixel of one “frame” in theimaging. The photoelectric charge representative of the next frame is insample and hold circuit 120 while the counter 140 is generating thedigital value representative of the next frame. The register 150 holdsthe pixel value until it is output as part of the image or for furtherprocessing. Each of the digital photocells that compose the imager pixelarray may be regulated using the same timing and control sequence sincethe photocells act in parallel, outputting an entire frame periodically.

[0017] To ensure that the dynamic range of the counter matches thedynamic range of the photocells, the sample and hold amplifier can beequipped to scale the input to VCO 120 as appropriate. The dynamic rangemay be mismatched due to differing ambient light levels in the scenebeing captured. The variance in integration period that may result froma charge in ambient light of the scene ensures that the captured imagehas the proper contrast. To adjust the dynamic range of the VCO 120 tomatch the analog photocell, a global scaling voltage 160 can be appliedto the sample and hold amplifier of each digital photocell in the arraywhich uniformly adapts the VCO component in each photocell to have adynamic range consistent with the present ambient light conditions. Theenhanced digital photocell of FIG. 1 may be utilized in a serial imagingdevice, or for use in parallel image processing architectures.

[0018]FIG. 2 is a simplified block diagram of a conventional serialimager.

[0019] As noted above, motion video compression such as MPEG, utilizes adifferencing approach to encode successive frames of image data. Aftertwo frames are captured or imaged completely, the difference betweencorresponding pixels is computed and this difference is then encoded.This allows highly correlated or redundant image features to be encodedusing the fewest number of bits. For instance, in a video-conferencingsession, the background of the captured image may change only slightlyor not at all from one frame instance to the next. In this case, ratherthan transmitting the entire background portion at a successive frame,just the pixel variance between frames could be transmitted. In FIG. 2,the serial imager utilized in CCD imaging devices would shift out pixelinformation for an entire frame and then another entire frame before thefirst difference frame could be computed. This conventional methodrequired thus the capturing and storing of two (or more) entire imageframes to generate a third frame representing the difference. The firstframe, a “key” frame is captured and digitized, as is a successiveframe. The digitized frames are then differed to generate a differenceframe. To reduce the delay and computational load in conventional imagedifferencing apparatus, an architecture similar to that of FIG. 3 may beemployed.

[0020]FIG. 3 illustrates an architecture for more efficient imagedifferencing. The conventional design of imaging devices is to performdifferencing of successive captured frames after they are captured anddigitized. To improve upon this conventional design, the computationalload can be reduced significantly if an analog differencing is doneprior to digitizing. The parallel-shift differencing apparatus of FIG. 3utilizes shifting to achieve the goal of generating image differencedata.

[0021] Consider a set of eight exemplary analog photocells A₃₁, A₃₂,A₃₃, A₃₄, A₃₅, A₃₆, A₃₇, and A₃₈. Photocells capture analog lightintensity at fixed locations in the scene. These intensity values arerepresented by an amount of charge that accumulates in the photocellduring its integration time. The photocells A₃₁, . . . , A₃₈ do notgenerate a digitized output as does the digital photocell unit ofFIG. 1. Rather, the stored charge is passed at the end of theintegration period (which is the same for all photocells in a givenframe instant), to a corresponding shift cell. For each row of analogphotocells, there are two rows of shift cells. One row of shift cellsstores photocell outputs for a first frame (“key” frame) while a secondrow of shift cells stores the photocell outputs of the successive frame.Each row of shift cells outputs photocell data serially.

[0022] The row of shift cells for key frame output are designated S₃₂,S₃₄, S₃₆, S₃₈, S₄₀, S₄₂, S₄₄ and S₄₆. The row of shift cells storingoutput for the frame immediately succeeding the key frame are labeledS₃₁, S₃₃, S₃₅, S₃₇, S₃₉, S₄₁, S₄₃ and S₄₅. When the imaging architectureis first initialized, all shift cells store a null or zero intensityvalue. When the first image frame is captured, each of the analogphotocells A₃₁, . . . , A₃₈ will develop a charge representative oflight intensity at a particular location in the scene that is incidentupon the photocell. This set of signals is transferred to the row ofshift cells S₃₁, S₃₃, . . . , S₄₆. The row of shift cells for key frameis at that instant, unfilled. Rather than outputs this first frame ofdata, the architecture waits until the next frame is captured. When thenext image frame is captured by analog photocells A₃₁, . . . , A₃₈, theresult of the previous frame is first transferred from shift cells S₃₁,S₃₃, . . . , S₄₅ to the row of shift cells S₃₂, S₃₄, . . . , S₄₆,respectively, as indicated in FIG. 3. Next, at the end of theintegration period for the second frame, the signals are transferredfrom analog photocells A₃₁, . . . , A₃₈ to the shift cells S₃₁, S₃₃, . .. , S₄₅. At that instant, both rows of shift cells contain image frameinformation. The row of shift cells S₃₂, S₃₄, . . . , S₄₆ which storesthe first frame is shifted out. This represents a key frame output 312.Key frame output 312 is simultaneously shifted to an input differentialop-amp (operational amplifier) 310.

[0023] The result of the current frame stored in shift cells S₃₁, S₃₃, .. . , S₄₅, is shifted out to the other input of differential op-amp 310.Differential op-amp 310 generates an analog signal, delta frame output314, which is the result of previous frame (key frame) subtracted fromthe current frame. The analog signal delta frame output 314 and keyframe output 312 may both be digitized prior to storage or processing.Since a serial shifting operation will output the analog key frame andcurrent frame outputs only pixel by pixel, the entire current frame andkey frame must first be shifted to output and to differential op-amp310. After the serial shifting operation is complete and the last of thekey and current frames are output/processed, then the current framestored in shift cells S₃₁, S₃₃, . . . , S₄₅ is shifted in parallel tothe row S₃₂, S₃₄, . . . , S₄₆ and thus, becomes the next key frame.

[0024] The advantages of this design lie primarily in the ability tosend to digitization only an analog difference frame output, rather thantwo entire image frames. Depending on the further down-the-lineprocessing to be performed, the delta frame output 314 and/or the keyframe output 312 may be digitized. In the conventional design, twoentire frames of analog photocell information is captured, and shiftedout separately, after which digitizing and differencing are performed.In the architecture of FIG. 3, both the key frame 312 and thedifferential for the next frame (delta frame output 314) are shifted tooutput simultaneously.

[0025] The embodiment of FIG. 3 still requires the shifting out of anentire key frame and difference frame, albeit simultaneously, beforeanother frame can be captured by the analog photocell. A furtherimprovement to this architecture is shown in FIG. 4 according to yetanother embodiment of the invention.

[0026]FIG. 4 illustrates one embodiment of the invention.

[0027] In the embodiment of FIG. 4, the current frame is shifted outimmediately and is also regenerated and fed back by way of an op-amp420.

[0028] In the architecture of FIG. 4, a current image frame is capturedby an array of N+1 analog photocells A₀, A₁, . . . , A_(N) which arethen passed in parallel to shift cells C₀, C₁, . . . . , C_(N),respectively. Shift cells C₀, C₁, . . . , C_(N) shift out the capturedframe data in a cascade (bucket-brigade) fashion from C_(N) to C₀. Thecurrent frame data is regenerated by an op-amp 420 and fed back to anarray of shift cells S_(N), . . . , S₀ as shown. As the regeneratedcurrent frame is fed back, the current frame is differenced against aprevious frame shifted out of the array of shift cells S₀, . . . ,S_(N). The differencing between the current frame and the previous frameis accomplished by an op-amp 410 and produces a pixel-by-pixeldifference frame output. This serial imaging system has the intendedadvantage of providing both the current frame and a difference framewithout having to wait for an entire frame of pixel data to be captured.Ordinarily, two frames, a first frame and a second frame must both becaptured before a difference frame can be generated. The architecture ofFIG. 4 eliminates such a limitation on serial imaging. Op-amps 410 and420, though not described in detail, can be designed by one of skill inthe art but should have the capabilities of boosting signal integrity inthe case of op-amp 420 and differencing two signals in the case ofop-amp 410.

[0029]FIG. 5 illustrates a per-pixel analog difference engine accordingto an embodiment of the invention.

[0030] The embodiment of FIG. 5 allows the transmission of multipledifference frames based upon a key frame. The imaging apparatus wouldfirst capture and transmit at the image output the key frame where noneof the pixels are differential. This key frame, shifted out via shiftcells S₅₁, S₅₂, . . . , S₅₄ is also fed back into an array of analogholding registers H₅₁, H₅₂, . . . , H₅₄. Prior to being fed back andoutput, each pixel is passed to a regeneration amplifier R5 whichregenerates the charge level of the pixel to avoid loss in the delay oftransmission. This feedback of the current frame into holding registersH₅₁, . . . , H₅₄ assures that the frame will be available as the“previous” frame when the next frame is captured. With the completedframe thus stored, it is possible to calculate a difference frame andtransmit this difference at the next frame cycle.

[0031] To achieve this, the output each analog holding register H₅₁,H₅₂, H₅₃ and H₅₄ is linked to the input of a differential operationalamplifier O₅₁, O₅₂, O₅₃ and O₅₄, respectively. The current frame iscaptured by analog photocells A₅₁, A₅₂, A₅₃ and A₅₄ whose output ispassed to other input of differential operational amplifier O₅₁, O₅₂,O₅₃ and O₅₄, respectively. After the key frame is transmitted, at theoutput, each subsequent frame may be transmitted as the differencerelative to the previous frame or key frame as computed by amplifiersO₅₁, O₅₂, O₅₃ and O₅₄. Any number of subsequent “difference” frames maybe transmitted to the output until the next key frame is desired. Theinput to each shift cell S₅₁, S₅₂, S₅₃ and S₅₄ is one of either theentire original frame captured by analog cells A₅₁, A₅₂, A₅₃ and A₅₄,respectively, (when a key frame is desired) or the difference output ofthe operational amplifiers O₅₁, O₅₂, O₅₃ and O₅₄, respectively.

[0032] A select signal (not shown) is sent to each of a set of analogmultiplexers M₅₁, M₅₂, M₅₃ and M₅₄ which routes either the appropriatekey frame data (frame A₅₁, A₅₂, A₅₃ and A₅₄, respectively) or differenceframe data (O₅₁, O₅₂, O₅₃ and O₅₄) as desired by the application user toshift cells S₅₁, S₅₂, S₅₃ and S₅₄, respectively. Additional rows ofshift cells and similar architecture may be linked together one afteranother as described.

[0033]FIG. 6 illustrates a per-pixel digital difference engine accordingto an embodiment of the invention.

[0034] The embodiment of FIG. 6 allows the transmission of multipledifference frames based upon a key frame. The imaging apparatus wouldfirst capture and transmit at the image output the key frame where noneof the pixels are differential. The initial key frame which is capturedby digital photocells (pixels) D61, D62, D63 and D64, is output onoutput bus 600. Simultaneously, the digital pixels D61, D62, D63 and D64is fed to a series of digital holding registers H61, H62, H63 and H64,respectively. This feedback of the current frame into holding registersH₆₁, . . , H₆₄ assures that the frame will be available as the“previous” frame when the next frame is captured. With the completedframe thus stored, it is possible to calculate a difference frame andtransmit this difference at the next frame cycle.

[0035] To achieve this, the output each digital holding register H₆₁,H₆₂, H₆₃ and H₆₄ is linked to the input of a subtraction unit S₆₁, S₆₂,S₆₃ and S₆₄, respectively. The current frame is captured by digitalphotocells D₆₁, D₆₂, D₆₃ and D₆₄ whose output is passed to other inputof subtraction unit S₆₁, S₆₂, S₆₃ and S₆₄, respectively. After the keyframe is transmitted, at the output, each subsequent frame may betransmitted as the difference relative to the previous frame as computedby subtraction units S₆₁, S₆₂, S₆₃ and S₆₄. Any number of subsequent“difference” frames may be transmitted to the output until the next keyframe is desired.

[0036] The output bus transmits one of either the key frame pixels (fromD₆₁, D₆₂, D₆₃ and D₆₄) or difference frame pixels (from subtractionunits S₆₁, S₆₂, S₆₃ and S₆₄) depending on what the applicant/userdesires. Based upon the desired mode, key or difference, a select signal(not shown) is sent to each one of digital multiplexers M₆₁, M₆₂, M₆₃and M₆₄ which then accordingly routes either key frame pixels (from D₆₁,D₆₂, D6 ₃ and D₆₄, respectively) or difference frame pixels (from S₆₁,S₆₂, S₆₃ and S₆₄, respectively) as indicated. Additional digital outputssimilar to those provided by M₆₁, M₆₂, M₆₃ and M₆₄ may be repeatedlyconstructed for each pixel desired.

[0037] The exemplary embodiments described herein are provided merely toillustrate the principles of the invention and should not be construedas limiting the scope of the invention. Rather, the principles of theinvention may be applied to a wide range of systems to achieve theadvantages described herein and to achieve other advantages or tosatisfy other objectives as well.

What is claimed is:
 1. An apparatus comprising: an analog photocelladapted to capture light energy incident upon it as an analog signal; asample-and-hold amplifier coupled to said photocell and adapted to storesaid analog signal; a digital converter coupled to said amplifier saidconverter transforming said analog signal into a digital value, saidvalue proportional to the amount of said light energy.
 2. An apparatusaccording to claim 1 wherein said digital converter includes: a voltagecontrolled oscillator; a counter coupled to said oscillator, saidoscillator setting the rate of increase of said counter, said rateproportional to said stored analog signal.
 3. An apparatus according toclaim 2 further comprising: a register coupled to said counter, saidregister receiving said digital value as an output of said counter atthe end of a predetermined time period.
 4. An apparatus according toclaim 2 , wherein said digital converter includes: a scaling signalsupply, said supply adapting the output of said oscillator in a dynamicrange consistent with ambient lighting to which said photocell isexposed.
 5. An apparatus according to claim 1 utilized in an imagingdevice.
 6. A system comprising: an array of analog photocells; a firstarray of shift cells, each of said first array shift cells coupled toone of said analog photocells; and a second array of shift cells coupledto said first array shift cells such that each first array shift cell iscoupled to one of said second array shift cells.
 7. A system accordingto claim 6 further comprising: a differential operational amplifierhaving two input terminals, one input terminal coupled to theterminating output of said first array of shift cells, the other inputterminal coupled to the terminating output of said second array of shiftcells, said amplifier providing a signal representative of thedifference between said first array terminating output and said secondarray terminating output.
 8. A system according to claim 7 , wherein aset of such signals, said set as large as the size of said first array,represent a delta frame of an image.
 9. A system according to claim 7 ,wherein said second array terminating output represents a key frame ofan image when said system is first initiated.
 10. A system comprising: afirst array of shift cells, the output of each of said first array shiftcells coupled to the input of the next of said first array shift cells;a second array of shift cells; and an array of analog photocells, eachof said photocells coupled to a corresponding one of said second arrayshift cells.
 11. A system according to claim 10 comprising: adifferential operational amplifier, having two input terminals, oneinput terminal coupled to the terminating output of said first array ofshift cells, the other input terminal coupled to the terminating outputof said second array of shift cells, said amplifier providing a signalrepresentative of the difference between said first array terminatingoutput and said second array terminating output.
 12. A system accordingto claim 11 , wherein a set of such signals, said set as large as thesize of said first array, represent a delta frame of an image.
 13. Asystem according to claim 11 further comprising: a regenerationamplifier having an input terminal coupled to the terminating output ofsaid second array of shift cells, the output of said regenerationamplifier coupled to the initiating input of said first array of shiftcells, said regeneration amplifier enhancing the terminating output ofsaid second array of shift cells.
 14. An apparatus comprising: a digitalphotocells, representing the light intensity of an area of an image as apixel value; a holding register coupled to said photocell, said registerreceiving said value; and a subtraction unit coupled to both saidphotocell and said holding register, the subtraction unit differencing acurrent pixel value of said photocell with a previous pixel value asstored in said holding register.
 15. An apparatus according to claim 14further comprising: an output bus; and a multiplexer coupled to saidsubtraction unit and said digital photocell, said multiplexerselectively providing one of the output of said subtraction unit and thevalue in said digital photocell to said output bus.